필요 라이브러리 설치¶
- scorecardpy lib install : https://github.com/ShichenXie/scorecardpy
- 변수 binning 및 scorecard 생성 library
- lightgbm lib install : https://lightgbm.readthedocs.io/en/latest/
- Boosting 알고리즘으로 모형 성능이 높고 학습 속도가 빠른 leaf wise 형태의 앙상블 알고리즘
In [ ]:
pip install scorecardpy
In [ ]:
pip install lightgbm
사전 환경 설정¶
- ./input 폴더에 학습 데이터 저장
Library Import¶
In [120]:
# Library Import
import os
import gc
import random
import pandas as pd
import numpy as np
from sklearn.metrics import roc_auc_score
import xgboost as xgb
import lightgbm as lgb
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import LabelEncoder
import missingno as msno
import plotly.express as px
import warnings
from sklearn.preprocessing import MinMaxScaler, StandardScaler
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression, Lasso
import scorecardpy as sc
from plotly.offline import init_notebook_mode, plot
init_notebook_mode()
# igore warning message
warnings.filterwarnings("ignore")
# pandas display option setting
pd.set_option('display.max_columns', None)
%matplotlib inline
1. Data Load ¶
File List Check¶
In [2]:
# File List Check
print(os.listdir("./input"))
Data Load¶
In [3]:
app_test = pd.read_csv('./input/application_test.csv')
app_train = pd.read_csv('./input/application_train.csv')
bureau = pd.read_csv('./input/bureau.csv')
bureau_bal = pd.read_csv('./input/bureau_balance.csv')
cc_bal = pd.read_csv('./input/credit_card_balance.csv')
cash_bal = pd.read_csv('./input/POS_CASH_balance.csv')
pre_app = pd.read_csv('./input/previous_application.csv')
ins_pay = pd.read_csv('./input/installments_payments.csv')
In [4]:
# Data Shape Check
print (app_test.shape, app_train.shape, bureau.shape, bureau_bal.shape, cc_bal.shape, cash_bal.shape, pre_app.shape, ins_pay.shape)
2.1. Target(Default) EDA(생략-Target에 대한 정의 및 데이터가 없기 때문에 수행 불가)¶
- 전이율 분석(roll rate 분석)
- 30일 연체 -> 60일 연체 -> 90일 연체로 넘어가는 비율 분석
- Performance Period 분석
- 9개월 -> 12개월 -> 15개월 -> 18개월 -> 24개월 기간의 Default의 증가 추이 분석
In [8]:
# Train / Test Data Merge
app_train['DataCategory'] = 'TRAIN'
app_test['DataCategory'] = 'TEST'
all_app = pd.concat([app_train, app_test], axis=0)
print (all_app.shape)
In [9]:
"""
# Category Type Column Name
"""
cat_cols = [col for col in all_app.select_dtypes(include='object').columns if col not in ['DataCategory']]
print (cat_cols)
In [10]:
print ('### Category Column Unique Values ###')
print ('-'*100)
for col in cat_cols:
print ('%s : %s'%(col, list(all_app[col].unique())))
In [11]:
"""
# 데이터 요약 정보 확인
"""
all_app.describe()
Out[11]:
In [13]:
all_app[all_app['DAYS_EMPLOYED'] >0]['DAYS_EMPLOYED'].describe()
Out[13]:
- Application data의 DAYS_EMPLOYED의 경우 대출 실행일 이전에 고용일수를 나타내는 항목인데 365243 양수 값이 존재함
- 64648건의 양수 값은 별도의 오류 데이터로 추정되어 nan값으로 변환
In [14]:
all_app['DAYS_EMPLOYED'].replace(365243, np.nan, inplace= True)
2.2.2. Feature Missing Value Check¶
- Application Data의 경우 120개 변수 중 67개 변수에 Missing Value가 존재
In [15]:
# Missing Value 존재하는 컬럼
missing_cols = app_train.columns[app_train.isnull().any()].tolist()
print ('## Missing Value # of Columns : %s'%len(missing_cols))
print (missing_cols)
- COMMONAREA_MODE 컬럼의 경우 약 70%가 Missing 값으로 구성되어 있음(최대 Missing 컬럼)
In [16]:
app_train[missing_cols].isnull().sum().reset_index().rename(columns={0:'missing_cnt'}).sort_values('missing_cnt', ascending=False)[:1]
Out[16]:
In [18]:
"""
# Missing Value Count Check
"""
msno.bar(app_train[missing_cols], figsize=(16, 5), color='#34495e',labels=True, fontsize=8)
Out[18]:
In [20]:
"""
# Data Completeness Check
"""
msno.matrix(app_train[missing_cols], figsize=(14, 5), labels=True, fontsize=8)
Out[20]:
- ata Completeness : 최대 67개, 최소 6개의 컬럼의 값이 존재(61개 컬럼 Null)하는 Row로 구성되어 있음
In [121]:
default_agg = {
'TARGET' : ['sum', 'count']
}
In [122]:
cont_type_agg = app_train.groupby(['NAME_CONTRACT_TYPE']).agg({**default_agg}).reset_index()
cont_type_agg.columns = ['_'.join(col).strip() for col in cont_type_agg.columns.values]
cont_type_agg['TARGET_ratio'] = cont_type_agg['TARGET_sum']/cont_type_agg['TARGET_count'] * 100
cont_type_agg['CRatio'] = cont_type_agg['TARGET_count']/cont_type_agg['TARGET_count'].sum() * 100
cont_type_agg.head()
Out[122]:
In [123]:
def_ratio = list((cont_type_agg['TARGET_ratio'].astype(str).values))
loan_type = list(cont_type_agg['NAME_CONTRACT_TYPE_'].values)
labels = []
for (item1, item2) in zip(loan_type, def_ratio):
labels.append(item1 + '(Default Ratio:' + item2[:3] + '%)')
In [127]:
fig = px.pie(cont_type_agg, values='CRatio', names=labels, title='Product Component Ratio')
fig.show()
- Revolving Loan 상품(부도율 5.4%)의 경우 최소 약정비율만 상환하면 자동으로 이월 되기 때문에 Cash Loan 상품(부도율 8.3%) 대비 부도율이 낮게 나타나고 있음
In [100]:
col_name = 'CODE_GENDER'
agg_df = app_train.groupby([col_name]).agg({**default_agg}).reset_index()
agg_df.columns = ['_'.join(col).strip() for col in agg_df.columns.values]
agg_df['TARGET_ratio'] = np.round(agg_df['TARGET_sum']/agg_df['TARGET_count'] * 100, 2)
agg_df['CRatio'] = np.round(agg_df['TARGET_count']/agg_df['TARGET_count'].sum() * 100, 4)
agg_df.head()
Out[100]:
- CODE_GENDER 값이 XNA인 4건의 경우 오류 데이터일 가능성이 높기 때문에 학습 데이터에서 제거 검토 필요
In [101]:
def_ratio = list((agg_df['TARGET_ratio'].astype(str).values))
loan_type = list(agg_df[col_name+'_'].values)
labels = []
for (item1, item2) in zip(loan_type, def_ratio):
labels.append(item1 + '(Default Ratio:' + item2 + '%)')
In [102]:
fig = px.pie(agg_df, values='CRatio', names=labels, title='Gender Component Ratio')
fig.show()
- 대출 고객은 남성이 65.8%, 여성이 34.2%, 알수없음(XNA)가 0.0013%를 차지하고 있음
- 부도율은 남성(10.14%)이 여성(7.0%)보다 3.14포인트 높게 나타남
In [103]:
col_name = 'FLAG_OWN_CAR'
agg_df = app_train.groupby([col_name]).agg({**default_agg}).reset_index()
agg_df.columns = ['_'.join(col).strip() for col in agg_df.columns.values]
agg_df['TARGET_ratio'] = np.round(agg_df['TARGET_sum']/agg_df['TARGET_count'] * 100, 2)
agg_df['CRatio'] = np.round(agg_df['TARGET_count']/agg_df['TARGET_count'].sum() * 100, 4)
agg_df.head()
Out[103]:
In [104]:
col_name = 'FLAG_OWN_REALTY'
agg_df = app_train.groupby([col_name]).agg({**default_agg}).reset_index()
agg_df.columns = ['_'.join(col).strip() for col in agg_df.columns.values]
agg_df['TARGET_ratio'] = np.round(agg_df['TARGET_sum']/agg_df['TARGET_count'] * 100, 2)
agg_df['CRatio'] = np.round(agg_df['TARGET_count']/agg_df['TARGET_count'].sum() * 100, 4)
agg_df.head()
Out[104]:
- FLAG_OWN_CAR, FLAG_OWN_REALTY 컬럼의 경우 각각 1.3포인트, 0.6포인트로 차 or 집의 보유 여부가 Default와는 큰 상관이 없음을 나타냄
In [105]:
col_name = 'CNT_CHILDREN'
agg_df = app_train.groupby([col_name]).agg({**default_agg}).reset_index()
agg_df.columns = ['_'.join(col).strip() for col in agg_df.columns.values]
agg_df['TARGET_ratio'] = np.round(agg_df['TARGET_sum']/agg_df['TARGET_count'] * 100, 2)
agg_df['CRatio'] = np.round(agg_df['TARGET_count']/agg_df['TARGET_count'].sum() * 100, 4)
agg_df.head()
Out[105]:
In [106]:
def_ratio = list((agg_df['TARGET_ratio'].astype(str).values))
loan_type = list(agg_df[col_name+'_'].astype(str).values)
labels = []
for (item1, item2) in zip(loan_type, def_ratio):
labels.append(item1 + '(Default Ratio:' + item2 + '%)')
In [107]:
fig = px.pie(agg_df, values='CRatio', names=labels, title='# of Children Component Ratio')
fig.show()
- 자녀수가 3명 초과의 경우 1% 미만의 구성비를 가지고 있기 때문에 3명 이상은 하나의 카테고리로 분류 검토 필요
- 자녀수가 증가함에 따라 부도율은 7.71%(0명)에서 9.63%(3명)로 1.9포인트 상승함
In [108]:
col_name = 'NAME_TYPE_SUITE'
agg_df = app_train.groupby([col_name]).agg({**default_agg}).reset_index()
agg_df.columns = ['_'.join(col).strip() for col in agg_df.columns.values]
agg_df['TARGET_ratio'] = np.round(agg_df['TARGET_sum']/agg_df['TARGET_count'] * 100, 2)
agg_df['CRatio'] = np.round(agg_df['TARGET_count']/agg_df['TARGET_count'].sum() * 100, 4)
agg_df = agg_df.sort_values('TARGET_ratio')
agg_df.head()
Out[108]:
In [109]:
def_ratio = list((agg_df['TARGET_ratio'].astype(str).values))
loan_type = list(agg_df[col_name+'_'].astype(str).values)
labels = []
for (item1, item2) in zip(loan_type, def_ratio):
labels.append(item1 + '(Default Ratio:' + item2 + '%)')
In [110]:
fig = px.pie(agg_df, values='CRatio', names=labels, title='NAME_TYPE_SUITE Component Ratio')
fig.show()
- 대출 신청시 가족이 같이 동행한(Family or Children) 고객의 경우 부도율이 7.49%, 7.38%로 다른 카테고리 대비 부도율이 낮음
In [111]:
col_name = 'NAME_INCOME_TYPE'
agg_df = app_train.groupby([col_name]).agg({**default_agg}).reset_index()
agg_df.columns = ['_'.join(col).strip() for col in agg_df.columns.values]
agg_df['TARGET_ratio'] = np.round(agg_df['TARGET_sum']/agg_df['TARGET_count'] * 100, 2)
agg_df['CRatio'] = np.round(agg_df['TARGET_count']/agg_df['TARGET_count'].sum() * 100, 4)
agg_df = agg_df.sort_values('TARGET_ratio')
agg_df.head()
Out[111]:
In [112]:
def_ratio = list((agg_df['TARGET_ratio'].astype(str).values))
loan_type = list(agg_df[col_name+'_'].astype(str).values)
labels = []
for (item1, item2) in zip(loan_type, def_ratio):
labels.append(item1 + '(Default Ratio:' + item2 + '%)')
In [113]:
fig = px.pie(agg_df, values='CRatio', names=labels, title='%s Component Ratio'%col_name)
fig.show()
- Working 그룹이 51.6%의 구성비로 가장 높은 구성비를 나타내지만, 부도율은 9.59%로 평균 부도율 8%대비 1.59포인트 높게 나타남
- 소득이 안정적인 연금수령자(Pensioner), 공무원(State servant)는 부도율이 각 5.39%, 5.75%로 낮게 나타남
- 소득이 불안정한 실업자(Unemployed), 출산휴가(Maternity leave)는 부도율이 36.36%, 40.00%로 매우 높게 나타남
In [114]:
col_name = 'NAME_EDUCATION_TYPE'
agg_df = app_train.groupby([col_name]).agg({**default_agg}).reset_index()
agg_df.columns = ['_'.join(col).strip() for col in agg_df.columns.values]
agg_df['TARGET_ratio'] = np.round(agg_df['TARGET_sum']/agg_df['TARGET_count'] * 100, 2)
agg_df['CRatio'] = np.round(agg_df['TARGET_count']/agg_df['TARGET_count'].sum() * 100, 4)
agg_df = agg_df.sort_values('TARGET_ratio')
agg_df.head()
Out[114]:
3. Simple Feature Engineering ¶
- Application Score 개발에 많이 활용되는 Application 정보 및 Bureau 정보를 활용한 간단한 Feature Engineering 수행
In [7]:
"""
# Label encoding
"""
def label_encoding(df, cat_cols):
le = LabelEncoder()
#cat_cols = [col for col in df.select_dtypes(include='category').columns]
for col in cat_cols:
df[col] = df[col].fillna('unknown')
df[col+'_le'] = le.fit_transform(df[col])
return df
In [40]:
"""
# Application Data Label Encoding
"""
all_app_label = label_encoding(all_app, cat_cols)
all_app_label.head()
Out[40]:
In [41]:
print (all_app_label.shape)
In [42]:
# Category Features drop
all_app_label.drop(cat_cols, axis = 1, inplace=True)
print (all_app_label.shape)
- 이전 대출 건수 변수 생성
- Missing Value의 경우 대출 건수가 0건
In [45]:
bureau.head()
Out[45]:
In [43]:
"""
# 이전 대출 건수 변수 생성
"""
previous_loan_counts = bureau.groupby('SK_ID_CURR', as_index=False)['SK_ID_BUREAU'].count().rename(columns = {'SK_ID_BUREAU': 'previous_loan_counts'})
previous_loan_counts.head()
Out[43]:
In [44]:
"""
# Bureau Data의 count, mean, max, min, sum, std 변수 생성
"""
bureau_agg = bureau.drop(columns = ['SK_ID_BUREAU']).groupby('SK_ID_CURR', as_index = False).agg(['count', 'mean', 'max', 'min', 'sum', 'std']).reset_index()
bureau_agg.head()
Out[44]:
In [46]:
columns = ['SK_ID_CURR']
"""
# Bureau 변수 Column Name 생성 Bureau Column name + count/mean/max/min/sum/std
"""
for var in bureau_agg.columns.levels[0]:
if var != 'SK_ID_CURR':
for stat in bureau_agg.columns.levels[1][:-1]:
columns.append('bureau_%s_%s' % (var, stat))
In [47]:
# Column Name Change
bureau_agg.columns = columns
bureau_agg.head()
Out[47]:
In [48]:
# Feature Engineering Result Merge
all_df = all_app_label.merge(previous_loan_counts, on = 'SK_ID_CURR', how = 'left')
all_df = all_df.merge(bureau_agg, on = 'SK_ID_CURR', how = 'left')
all_df.head()
Out[48]:
4.Feature Selection¶
- ML 알고리즘 기반 CSS 모형 개발시 변수 선택 방법
- Recursive Feature Elimination(RFE)
- 모형 성능에 영향을 적게주는 변수를 제거하면서 선택하는 방법
- Embeded Logic
- RandomForest, Xgboost, Lightgbm, Catboost 알고리즘의 Feature Importance 활용
- Filter 방법
- correlation, chi2 등의 통계값을 기준으로 필터링하여 선택하는 방법
- Recursive Feature Elimination(RFE)
- Score Card 기반 CSS 모형 개발시 변수 선택 방법
- Weight of evidence(WOE)와 Information Value 기반의 변수 선택 방법
- https://www.listendata.com/2015/03/weight-of-evidence-woe-and-information.html
4.1. RFE Function을 사용하여 변수 선택¶
- 다중공선성을 고려하여 변수 선택(LASSO)
- LASSO Normalization이 필요함
- Missing Value 처리 필요함
In [49]:
"""
# 전체 Feature List
"""
feats = [col for col in all_df.columns if col not in ['SK_ID_CURR', 'TARGET', 'DataCategory']]
print (len(feats))
In [50]:
"""
# Missing Value를 mean값으로 변환
"""
all_df.fillna(all_df.mean(), inplace=True)
In [51]:
"""
# Data Split & Normalization
"""
X_train = all_df[all_df['DataCategory']=='TRAIN'][feats]
X_test = all_df[all_df['DataCategory']=='TEST'][feats]
y_train = all_df[all_df['DataCategory']=='TRAIN']['TARGET']
y_test = all_df[all_df['DataCategory']=='TEST']['TARGET']
# Normalization
scaler = StandardScaler()
scaler.fit(all_df[feats])
X_train_norm = scaler.transform(X_train)
X_test_norm = scaler.transform(X_test)
print (len(X_train_norm), len(X_test_norm))
In [53]:
rfe_selector = RFE(estimator=Lasso(alpha=0.01, max_iter=3000), n_features_to_select=100, step=10, verbose=5)
rfe_selector.fit(X_train_norm, y_train)
Out[53]:
In [54]:
rfe_support = rfe_selector.get_support()
rfe_feature = X_train.loc[:,rfe_support].columns.tolist()
print(str(len(rfe_feature)), 'RFE Lasso selected features')
In [58]:
print ('-'*120)
print (rfe_feature)
4.2. Information Value(Weight of evidence)를 활용한 변수 선택¶
- Less than 0.02 : Not useful for prediction
- 0.02 to 0.1 : Weak predictive Power
- 0.1 to 0.3 : Medium predictive Power
- 0.3 to 0.5 : Strong predictive Power
- 0.5 over : Suspicious Predictive Power
In [62]:
"""
# Weight of evidence를 활용한 Binning
"""
bins = sc.woebin(all_df[all_df['DataCategory']=='TRAIN'], y="TARGET")
In [65]:
%%time
"""
# Information Value Check
"""
info_value = sc.iv(all_df[all_df['DataCategory']=='TRAIN'], y="TARGET", x=rfe_feature)
In [76]:
iv_feats = list(info_value[info_value['info_value']>=0.1]['variable'])
print (len(iv_feats), iv_feats)
In [67]:
"""
# Train/Test Data를 WOE Bin 값으로 변경
"""
train_woe = sc.woebin_ply(X_train[iv_feats], bins)
test_woe = sc.woebin_ply(X_test[iv_feats], bins)
In [68]:
%%time
"""
# LogisticRegression 모형 학습
"""
lr = LogisticRegression(penalty='l1', C=0.9, solver='saga', n_jobs=-1)
lr.fit(train_woe, y_train)
# predicted proability
train_pred = lr.predict_proba(train_woe)[:,1]
test_pred = lr.predict_proba(test_woe)[:,1]
print (roc_auc_score(y_train, train_pred))
In [77]:
"""
# 학습한 LogisticRegression 모형과 WOE binning 결과를 기준으로 Scorecard 생성
"""
card = sc.scorecard(bins, lr, train_woe.columns)
In [70]:
"""
# Scorecard Sample
"""
card['bureau_AMT_CREDIT_SUM_LIMIT_sum']
Out[70]:
In [71]:
%%time
"""
# Scorecard 기준 Score 계산
"""
train_score = sc.scorecard_ply(X_train[iv_feats], card, print_step=1)
test_score = sc.scorecard_ply(X_test[iv_feats], card, print_step=1)
In [72]:
"""
# 예측 등급의 안정성을 체크하기 위한 PSI(Population Stability Index) 지표 체크
"""
psi_chk = sc.perf_psi(
score = {'train':train_score, 'test':test_score},
label = {'train':y_train, 'test':y_test},
x_tick_break = 30, #등급별 점수 Range
return_distr_dat=True,
show_plot=False
)
In [73]:
"""
# PSI 지표 결과
- PSI < 0.1 : 안정
- 0.1 <= PSI < 0.25 : 다소 불안정
- PSI >= 0.25 : 불안성
"""
psi_chk['psi']
Out[73]:
In [74]:
psi_chk
Out[74]:
5. Hyperparameter Tunning¶
- Grid Search의 경우 Parameter 탐색 시간이 오래걸리기 때문에 Random Search로 구현
In [78]:
N_FOLDS=5
def objective(params, iteration):
"""
# Objective Function
"""
if 'num_boost_round' in params.keys():
del params['num_boost_round']
cv_results = lgb.cv(params, dtrain, num_boost_round = 1000, nfold=N_FOLDS, early_stopping_rounds=50, metrics=['auc'], seed=2020)
score = cv_results['auc-mean'][-1]
estimators = len(cv_results['auc-mean'])
params['num_boost_round'] = estimators
return [score, params, iteration]
In [81]:
random.seed(2020)
MAX_EVALS = 100
def random_search(param_grid, max_evals=MAX_EVALS):
"""
# Random Search for Hyperparameter optimization
"""
results=pd.DataFrame(columns=['score', 'params', 'iteration'], index=list(range(max_evals)))
for i in range(max_evals):
# Choose Random Params in param_grid
hyperparams = {k:random.sample(v,1)[0] for k,v in param_grid.items()}
hyperparams['subsample'] = 1.0 if hyperparams['boosting_type']=='goss' else hyperparams['subsample']
eval_results=objective(hyperparams, i)
results.loc[i,:]=eval_results
results.sort_values('score', ascending=False, inplace=True)
results.reset_index(inplace=True)
return results
In [82]:
"""
# Lightgbm Hyperparameter Grid
"""
lgb_param_grid = {
'objective ' : ['binary'],
'boosting_type' : ['gbdt'],
'num_leaves' : list(range(20, 150)),
'learning_rate' : list(np.logspace(np.log10(0.005), np.log10(0.3), base=10, num=100)),
'subsample_for_bin' : list(range(2000, 10000, 2000)),
'min_child_samples' : list(range(20, 500, 5)),
'reg_alpha' : list(np.linspace(0, 1)),
'reg_lambda' : list(np.linspace(0, 1)),
'colsample_bytree' : list(np.linspace(0.6, 0.9, 10)),
'subsample' : list(np.linspace(0.5, 0.9, 10)),
'drop_rate': list(np.linspace(0.01, 0.05, 10))
}
In [83]:
# Data Type Change
dtrain = lgb.Dataset(data = X_train[rfe_feature], label=y_train)
In [84]:
%%time
print ('Running Hyperparameter Search...')
random_results = random_search(lgb_param_grid, 15)
In [85]:
random_results.head()
Out[85]:
In [86]:
# Hyperparameter Setting
sel_params = random_results['params'].values
sel_param = sel_params[0]
print (sel_param)
In [87]:
# Model Train
gbdt = lgb.train(sel_param, dtrain)
pred_train = gbdt.predict(X_train[rfe_feature])
pred_test = gbdt.predict(X_test[rfe_feature])
Feature Importance Plot¶
In [88]:
feat_importance = pd.DataFrame()
feat_importance['feature'] = rfe_feature
feat_importance['importance'] = gbdt.feature_importance()
feat_importance = feat_importance[feat_importance['importance']>0]
feat_importance.shape
Out[88]:
In [89]:
feat_importance
Out[89]:
In [90]:
plt.figure(figsize=(8, 10))
sns.barplot(x="importance", y="feature", data=feat_importance.sort_values(by="importance", ascending=False))
plt.title('LightGBM Features')
plt.yticks(size=8)
plt.tight_layout()
In [91]:
print ('Train ROC Score : %s'%roc_auc_score(y_train, pred_train))
In [92]:
submission = pd.read_csv('./input/sample_submission.csv')
submission['TARGET'] = pred_test
submission.to_csv('./input/submission.csv', index=False)
In [ ]: